1. Field of the Invention
The present invention relates generally to product readiness assessment testing, and in particular, to computer-implemented tools for use in such testing.
2. Description of the Related Art
The development process in any organization typically has the competing concerns of schedule and quality. On the one hand, in many industries such as the computer industry, short product life cycles and frequent advances in technology necessitate short product development cycles under a "first to market" strategy. On the other hand, there is a need to make a product "right the first time", as research shows that on the average external failure costs are often an order of magnitude more expensive than the corresponding internal costs for finding the same problem prior to release of the product.
Various reliability engineering processes and tools have been developed to assist in product reliability testing. For example, histograms, reliability demonstrations, and control charts have been developed to assist in the determination of whether a product under development has been proven to meet a desired reliability goal. For example, for computer software and hardware products, the desired reliability goal may be in units of Mean Time Before Failure (MTBF) or Service Calls/Machine/Year (SC/M/Y). Standard tests and demonstration methods for these products are still based on Military Standards developed during the World War II era.
One particularly useful test that has been developed is the sequential probability ratio test (SPRT), which may be used to determine when sufficient reliability testing has been performed to prove with a desired level of confidence that a product has met a reliability goal.
With the SPRT, data is typically accumulated manually during product testing relating to the number of failures of the product and the cumulative runtime of a product during testing. In addition, accept and reject lines are manually calculated based upon a desired reliability goal, a desired confidence level that the goal has been met, and a discrimination ratio that is related to the power of the test to detect various levels of reliability (e.g., a test with discrimination ratio of 2 is designed to accept product with MTBF=M and reject product with MTBF=M/2 with high probability). The accept and reject lines provide instantaneous values of acceptable and unacceptable numbers of failures for any given amount of runtime.
As product reliability testing is performed, the accumulated number of failures and runtimes are typically plotted and compared against the accept and reject lines. When the cumulative failures after a given runtime are between the accept and reject lines, additional testing is required. If the cumulative failures cross above the reject line (and into a "reject range"), the product reliability test for the product has failed, indicating that additional engineering, diagnosis, and/or risk assessment may need to be performed, with the test re-run at a later date.
However, if the cumulative number of failures crosses below the accept line after a given runtime (and into an "accept range"), the SPRT indicates that the desired reliability goal has been met with the desired level of confidence. Accordingly, at this point the product testing may be completed and the product released, e.g., to the customer, or to the next stage of development.
One advantage of the SPRT is that the relative performance of a product compared to the desired performance goals may be monitored over time to see how the product test is progressing. Thus, maintaining the cumulative failures between the accept and reject lines often provides the motivation to an organization to continue the product reliability test.
One problem, however, with the SPRT and other reliability testing methods is that often the time required to perform the full tests exceeds the allotted development time for the product. Particularly in the computer industry, the "first to market" goals often require management to release a product before the reliability testing has been completed. Often, management will not even utilize reliability testing methods such as the SPRT due to the expectation that the reliability test would take longer than is available. Often, therefore, management must make uninformed judgment calls as to when to release a product.
A strong need has existed for a manner of assisting management in the determination of when to release a product, particularly prior to completion of product reliability testing. Moreover, a strong need exists for providing a more quantitative assessment of the risks associated with such decisions, so that more objective decision making may occur.